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A growing literature supports cognitive therapy (CT) as an efficacious treatment for 
youth struggling with emotional or behavioral problems. Recently, work in this area has 
extended the dissemination of CT to school-based settings. The current study has two 
aims: 1) to examine the development of therapists’ knowledge and skills in CT, an 
evidence-based approach to promoting student well-being, and 2) to examine patterns of 
narrative feedback provided to therapists participating in the program. As expected, 
school therapists trained in CT demonstrated significant gains in their knowledge of CT 
theory and in their demonstration of CT skills, with the majority of therapists surpassing 
the accepted threshold of competency in CT. In addition, an examination of feedback 
content suggested that narrative feedback provided to therapists most frequently 
consisted of positive feedback and instructions for future sessions. Suggestions for future 
research regarding dissemination of CT are discussed in light of increasing broad access 
to evidence based practices. 
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Introduction 

Cognitive behavioral therapy (CBT) has been established as an effective approach to support the 
emotional, behavioral, and social needs of youth and to improve their functioning in school (Kazdin & Weisz, 
2010). However, successful implementation of CBT in school settings has lagged significantly behind the 
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establishment of that evidence (Creed, Reisweber & Beck, 2011; Forman & Barakat, 2011). Although the call 
for empirically supported school-based intervention has been steadily increasing in the current culture of 
evidence based practices (Forman & Burke, 2008), reports of successful dissemination to schools have 
remained limited. Most studies assessing dissemination or implementation of CBT in schools have targeted a 
very specific presenting problem, most frequently the prevention or reduction of substance abuse, aggression, 
or violence (Forman & Bakarat, 2011). However, these targeted interventions may not represent the breadth 
and complexity of presenting problems typically encountered in school settings. A more flexible approach 
may better prepare therapists to meet students’ diverse challenges, promote wellness, and help students move 
toward their individual goals. The goal of the current study was to evaluate the training outcome of this 
school-based initiative. Patterns of written feedback provided to therapist trainees were examined to explicate 
a mechanism through which the outcomes may have been achieved. This successful model of training may 
offer some guidance in the further uptake of CBT and other evidence based practices in the schools, 
increasing the accessibility of these interventions for children and adolescents. 

Since 2007, the Beck Initiative has endeavored to implement Cognitive Therapy (CT) in community 
mental health settings, increasing access to this empirically supported treatment by those who are served by 
community mental health. CT is specified to distinguish this specific approach from the broader term, CBT, 
signifying that the case conceptualization is used to guide intervention according to the cognitive model, as 
originally developed by Beck (Beck, Rush, Shaw & Emery, 1979). In recent years, services for children and 
adolescents have been a major focus of The Beck Initiative, with an emphasis on school-based services. 
Implementation of CT in schools delivers a powerful intervention for youth without increasing the burden on 
stressed families to attend outpatient treatment, reduces the loss of academic time to travel to outpatient 
appointments, eliminates transportation issues, and facilitates the integration and generalization of new skills 
into a student’s daily setting. Therefore, the Beck Initiative offers a unique opportunity for many underserved 
youth to benefit from access to evidence based practices and paves the way for the successful implementation 
of these practices in youth-accessible settings. 

Overview of the Beck Initiative Training Program 

Established in 2007, the Beck Initiative is a public-academic partnership in a northeastern American 
city. This innovative, team-oriented approach advances the quality of care provided to persons in recovery 
from behavioral health concerns within this community mental health care system by teaching clinicians 
tangible, empirically-based skills. Objectives of this program are to: (a) incorporate the evidence-based 
practice of CT as a standard of care within the network; (b) facilitate recovery and improve treatment 
outcomes for people served within this healthcare system; (c) improve the professional lives of therapists in 
the network; (d) conduct program evaluation to measure outcomes; (e) become one of the first large-scale 
implementations of evidence-based psychotherapy; and (f) serve as a model for other large behavioral health 
systems (Stirman, Buchhofer, McLaulin, Evans, & Beck, 2009). 

Participation in the Beck Initiative is intensive and requires a substantial commitment from 
the agencies and the therapist trainees. According to the established ACCESS training model (Stirman, Bhar, 


ISSN 2073-7629 
©2013 CRES/ENSEC 


Volume 5. Number 2, November 2013 


pp 50 



Spokas, Brown, & Creed, 2010), therapists within a participating agency are nominated for participation by 
their administration. The nominated therapists who elect to participate and meet basic participation 
requirements (have a current caseload, hold at least a master’s degree, demonstrate fluency in English) 
become identified trainees. Essential components of Beck Initiative training include participation in 22 hours 
of intensive workshop, 6 months of weekly consultation with therapy session tape review, evaluation of 
competency in CT based on the Cognitive Therapy Rating Scale (CTRS; Young & Beck, 1980), and ongoing 
internal supervision groups held within the agency. The workshop phase is completed in 4-5 weekly sessions 
and covers content including (a) the cognitive model, (b) case conceptualization, (c) cognitive interventions, 
(d) behavioral interventions, (e) tailoring the treatment approach to the individual, (f) stages of treatment and 
(g) competency and the CTRS. The 6-month consultation period was designed to maximize therapist benefit 
and structured to be consistent with empirical evidence that indicates quality of feedback is related to therapist 
competency (i.e. Brosnan, Reynolds, & Moore, 2006). Weekly meetings are held with the trainee group to 
review audio samples from the past week, share peer and trainer feedback about the audio, develop and refine 
case conceptualizations, brainstorm about creative ways to integrate CT approaches into the school milieu, 
and provide additional didactic information about special topics as requested by the trainees. After the 
workshop and 6 months of consultation, agencies are expected to continue to hold peer consultation meetings 
indefinitely to prevent drift from the model. Ongoing support is provided to the agency through instructor 
visits every 6-8 weeks and participation in quarterly meetings among all trained agencies within the network. 

The Beck Initiative training model (Stirman et al., 2010) emphasizes the importance of providing 
trainees with consistent, structured feedback over time. Formal written feedback is provided through narrative 
responses and ratings on each of the 11 items of the CTRS at the mid-point and end of the 6-month 
consultation period, and this written feedback is structured so as to be strength-based. Specifically, for each 
item of the CTRS, instructors provide detailed written feedback regarding (1) the strengths or skills 
demonstrated by the trainee in the session, and (2) ways in which the strengths can be built upon to increase 
fidelity and competency in CT. This model is consistent with the extant research suggesting that the structure, 
content, and process of supervision are all integral to providing effective supervision. The structure of 
supervision has been shown to have an impact on therapist satisfaction and effectiveness. Specifically, 
supervision that includes a structured agenda, audio review, and competency ratings has been suggested to be 
atypical, but desirable and effective (Cox & Araoz, 2009). Furthermore, group supervision has been shown to 
have a number of benefits, including increased support among professional peers leading to better 
communication, planning, and delivery of treatment (Alleyne & Jumaa, 2007). Group supervision also 
enhances the cost efficiency of training by maximizing the impact of supervision while limiting the cost, 
which is particularly important given the current and expanding fiscal pressure on schools, providers and 
payers. Finally, more frequent supervision has also been associated with positive outcomes, as researchers 
have found that the more time trainees spend in supervision, the more they value the supervision (Fivni, 
Crowe, & Gonsalvez, 2012). Notably, the structure of the group supervision in the Beck Initiative mirrors 
some of the CT approach with students, including an agenda, a collaborative stance, use of specific 
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constructive feedback, identification of skills to be practiced outside of the meetings and processing of the 
success of those practices. 

Although research on the effect of feedback on therapist skill development is quite limited, evidence 
suggests that supervision should include consistent feedback with an opportunity for collaboration between 
trainee and supervisor (Falender & Shafranske, 2007). In particular, supervision tends to be most effective 
when supervisors provide consistent positive feedback and invite feedback from trainees (ibid.). Positive 
feedback has also been associated with increasingly efficient therapy, as it can help reduce the erosion of 
fidelity. Furthermore, therapists in training tend to prefer supervisors who provide support, insight, and 
instruction (Kennard, Stewart, & Gluck, 1987). Finally, a limited body of research suggests that providing 
feedback to therapists learning to implement CT affects the acquisition of CT skills. Specifically, Brosnan et 
al. (2006) found that providing direct supervision in cognitive therapy is related to demonstrated skills on the 
CTRS, even on the item that measures interpersonal effectiveness, a markedly global therapeutic skill. 

In sum, establishing supervision as a structured but active process with experiential methods and tape 
review is a major component of guidelines suggested for evidence based clinical supervision (Milne & Reiser, 
2011), and is likewise emphasized in the Beck Initiative. In addition, instructors in the Beck Initiative strive to 
deliver feedback that is positively-framed, building on strengths demonstrated in sessions, and oriented 
towards the goals of the trainee and his or her clients. By doing so, instructors mirror CT processes like skill¬ 
building and goal-setting that therapists concurrently learn to implement with students. There is a dearth of 
research examining the specific content of feedback (e.g. strength-based statements, instructions for 
improvement) provided in this effective approach to training therapists (Stirman et al., 2010). Therefore, 
examining the content of supervisory feedback may begin to elucidate the process by which therapists are 
successfully trained to competency in CT. Understanding this process would be a step toward identifying the 
most effective approaches to developing competency in evidence based practices. 

The current study had two aims, namely to report the outcomes of a training program offered to 
therapists in school settings that emphasized an evidence-based approach to promoting student well-being, 
and to examine patterns of narrative feedback provided to therapists participating in the program. It was 
expected that therapists would demonstrate significant improvements over time both in knowledge of CT 
theory and in demonstration of CT skill, demonstrating that community-based therapists can be trained to 
deliver evidence based treatment with competency, within the real-world challenges of urban public schools. 
Competent delivery of these services would, in turn, create the opportunity for students to participate in these 
services in the setting where they already spend their days, as a step toward decreased disparity in access to 
care. This study is also the first of its kind to examine the content of the narrative feedback provided to 
school-based therapists, providing insight into a component of an effective CT dissemination process. Broader 
understanding of the components of effective dissemination may increase the uptake of CT and other 
evidence based care in the school settings, adding to increased student access to care. 

Method 

Participants 
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One agency was contracted to work at a public high school, while the other four treated primary 
school students. Of those four programs, two were specialized schools for students with disciplinary 
problems, one was a specialized classroom within a mainstream school, and one was a mainstream school that 
contracted for mental health services through the city’s managed health care system. Overall, the therapists 
working at these agencies provide a variety of individual and group therapy services to students with both 
internalizing (e.g. depression, anxiety, grief/loss, managing family stress), and externalizing (e.g., 
oppositional or defiant behaviors, anger management) problems. 

The sample consisted of 25 therapists employed at five different school-based agencies in a 
metropolitan city in the northeast United States. The sample was 88% female (n = 22). The majority (n = 23, 
92%) of the sample reported having received a master’s degree, while one participant reported having 
received a Ph.D. and one an M.D. Participants worked in a variety of fields including counseling (n = 10, 
40%), education (n = 1, 4%), psychiatry, (n = 1, 4%), psychology (n = 3, 12%), and social work (n = 10, 
40%). A subset of the sample (« = 13), responded to optional questions regarding their training background 
and prior experience with CBT. This sample reported a range in years since completing graduate training 
from 1 to 29 years (M = 7.31, SD = 7.8). Only two therapists identified their theoretical orientation as 
“behavioral” or “cognitive behavioral.” Of the 13 therapists who responded, two (15%) had learned about 
CBT from reading books or articles, three (23%) had exposure to CBT in a graduate level course, six (46%) 
had attended a workshop in CBT, and two (15%) had attended a workshop with follow-up support. 
Regardless of how much prior exposure to CBT they reported having, most (n = 12) therapists indicated 
knowing “only the basics” while one reported knowing “nothing” and no one reported knowing “a great deal” 
about CBT. All participants in this sample attended the workshop in its entirety and at least 80% of the 
weekly consultation sessions. Because this program was primarily a training initiative, program evaluation 
measures were collected voluntarily and completion of these measures varied based on the individual 
measure. Sample sizes are reported for each measure in its corresponding section below. 

All training for school based agencies was provided by postdoctoral fellows (n = 5) selected for their 
training and expertise in CT and experience working with youth, under the supervision of the lead author, an 
expert in the implementation of CT in community mental health settings and director of the Beck Initiative. 
Fellows received live supervision during the training meetings and additional individual weekly supervision 
of their delivery of training. Each agency was assigned 2 instructors for the duration of training from among 
the 5 postdoctoral fellows and the Beck Initiative Director. 

Adaptation of the Beck In itiative for School-based Programs 

Embracing advantages and tackling challenges of the school setting. Colocation of mental health 
services in a school setting brings unique therapeutic opportunities and challenges. The traditional Beck 
Initiative training described above was delivered to therapists in a way that highlighted opportunities and 
resolved challenges, while maintaining fidelity to the CT model. The structure and delivery of the training and 
consultation remained the same: 22 hours of workshop and 6 months of weekly consultation, followed by 
transition to a peer-led consultation group with periodic support from Beck Initiative instructors. The 
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emphasis in this training was specific to helping therapists take advantage of the opportunities presented by 
school-based services, while overcoming the inherent challenges. Rather than modifying CT, this approach 
embraced the flexibility inherent in CT by identifying the ways in which a good CT therapist (regardless of 
setting) uses the cognitive model and case conceptualization to tailor treatment for any client. 

School-based therapists have the opportunity to observe their clients interacting with peers, teachers 
and other school staff in settings with little (recess, lunch) or a great deal of structure (classrooms). These 
opportunities also become settings for intervention and practicing coping skills. Therapists and clients worked 
together to develop coping skills for use during the school day, planned specific practice assignments for the 
client to use the skill in a school situation, and then were able to meet again to discuss the success of the 
practice - all within the course of a school day. Further, school-based therapists have access to information 
about their students beyond just their challenges. Whereas a depressed student may report that “nothing good 
has happened all week,” a school-based therapist may have additional information about the student’s 
successes from teachers, peers, administrators or others. 

Along with these opportunities were challenges unique to the school setting. A recurring challenge 
was the disciplinarian role into which therapists were often placed. School staff frequently perceived 
therapists to be responsible for managing the moment-to-moment classroom behavior of their clients, and 
would expect therapists to remove students with problematic behavior from the classroom to discipline them. 
These expectations interfered with therapists’ ability to deliver individual and group therapy, both logistically 
and because of the toll taken on therapeutic relationship. Therapists were encouraged to use their new CT 
skills to conceptualize and empathize with these teacher perceptions, and then problem-solve collaboratively 
with the teachers to find a more feasible solution. 

Collaboration with teachers. Colocation in the school also provided the opportunity for therapists to 
collaborate with teachers for targeted interventions (intervention intended for specific students in the 
classroom) and broader classroom-level interventions. The first step in these collaborations was to share the 
rationale for intervention with the teacher, followed by planning, implementation, and evaluation. For 
example, a student who struggled with reading would ruminate about his abilities, often becoming so upset 
and frustrated during individual reading activities that he would begin to disrupt the classroom. The therapist 
identified this pattern of thoughts, feelings and behavior with the student, and together they created a plan to 
help him reduce and cope with his distress. The plan was written on a small piece of paper and taped to the 
student’s desk. The teacher was able to cue the student to use his plan as needed by pausing near his desk and 
placing her finger on the piece of paper, facilitating the generalization of new skills from therapy room to 
classroom. The generalization and applied success built, in turn, a sense of being able to cope, ultimately 
leading to greater resilience in other situations. 

Therapists and teachers also collaborated to create classroom-level interventions. The steps for 
catching, checking and changing an unhelpful or inaccurate thought (Creed et al., 2011; Granholm, McQuaid, 
Auslander, & McClure, 2004) were taught to all students in a classroom, and then a poster outlining the steps 
was hung prominently in the room. The teacher and therapist encouraged students to use the process and 
language of this skill during classroom interactions. For example, during a conflict between two students, 
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each was prompted to catch the thought he had at the beginning of the argument, check whether the thought 
was true or helpful, and move to a more helpful and accurate thought. Over time, this process became part of 
the classroom culture, and students could be heard cueing each other to catch, check and change their 
thoughts. The normalization of the use of CT skills and support for those skills in the child’s environment 
were key components in classroom interventions. 

Goal setting and Individualized Educational Programs. A hallmark of CT is the collaborative goal 
setting that takes place with clients. The therapist helps the child as needed to identify and quantify personal 
goals, the skills and strengths already possessed by the client, and the challenges or roadblocks standing in the 
way of the child meeting his or her goals. The goals, strengths, and needs are then translated into treatment 
goals that have meaning to the client, with the intent of supporting the client’s growth in his or her intended 
direction. Collaborative goal setting remains a central point in school-based cognitive therapy, and may 
perhaps be particularly relevant for students who were referred to treatment for issues that they do not see as 
valuable treatment goals (Creed & Diamond, 2001; DiGiuseppe, Linscott, & Jilton, 1996; Shirk & Russell, 
1998). For example, one adolescent student indicated that her goal was for her teacher to yell at her less often. 
The therapist and student worked to identify the situations in which her teacher most often yelled at her. 
Together they concluded that her teacher most often yelled when the student left her seat without permission 
or argued with other students. The therapist and student developed goals to reduce each of these behaviors by 
(1) understanding what triggered the behavior (“What thought goes through my head before I leave my seat / 
start to argue with a peer?”), (2) checking whether those thoughts were accurate or helpful for her, (3) 
identifying a more accurate or helpful thought that would make it easier for the student to engage in a 
different behavior, and (4) learning and practicing new coping behaviors to help the student behave 
differently in those situations. 

Individual Education Programs, or lEPs, are mandated by the Individuals with Disabilities Education 
Act of 2004, a law in the United States that governs how educational services are provided to students with 
disabilities. The plan is guided by the student’s identified needs (U.S. Department of Education, 2007) and 
focuses on addressing deficits. Collaborative goal setting provides an alternative to organizing treatment 
around deficits, and instead focuses on working toward the student’s own goals. Whereas IEP goal setting 
may be set without the child attending the meeting, CT goals are led by the child. The strategy for achieving 
the goals is organized around helping the child overcome obstacles to the goals, rather than disability. 

Case conceptualization. Another hallmark of cognitive therapy is the case conceptualization, which 
elucidates the ways in which a student’s life experiences have informed core beliefs, intermediate beliefs or 
assumptions, and coping behaviors. Those patterns of thinking and behavior influence how students perceive 
and react in given situations. Case conceptualization brings two powerful pieces to bear on treatment in 
schools. First, it provides a common way of understanding a student’s behavior which frames that behavior 
as making sense, given the way the student perceives the world, him or herself, and others. Case 
conceptualization also helps the student to recognize unhelpful or inaccurate beliefs and shift them, resulting 
in moving the student closer to his or her goals. When school staff and students approach a student’s behavior 
in this way, they may experience more empathy and less frustration in response to the behavior. 
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Therapists in the Beck Initiative learned ways to communicate with teachers about the cognitive 
model, thus facilitating teachers’ understanding of how a student’s beliefs might impact his or her reactions 
(emotions and behavior) in school situations. For example, the traditional case conceptualization (Beck, 1995) 
was simplified into a simple three-box design (History, Beliefs, Behavior) to communicate these connections 
to teachers who has not been trained in CT. This shared conceptualization aimed to decrease stigmatization of 
student’s school behavior as “troublesome” or “malicious” and rather conceptualize it as behavior that made 
sense, given the way the student sees the world, him or herself, and others. Shared understanding of the 
origins of behavior can increase empathy and understanding, ultimately increasing the likelihood that the 
behavior (and related beliefs and emotions) may be shifted to something more adaptive (Creed et al., 2011). 
For example, one student was receiving services because he frequently made loud comments in class, 
resulting in laughter and disruption of the classroom. The therapist and child worked together to identify that 
in the child’s past (“History”), she had been openly ridiculed by a different teacher for making mistakes in 
class. The child had developed a belief that if she made a mistake, she would be humiliated. She had also 
learned that if she said disrupted the class with something humorous, she could divert attention from any 
mistakes she might make (Beliefs). Those beliefs led her to make loud comments whenever she began to 
worry that she was making an academic mistake (Behavior). When this conceptualization was shared with the 
teacher, he developed empathy for the child and was willing to collaborate on a plan for the child to ask for 
help quietly, with an explicit agreement that the child would never be teased for doing so. When the child 
agreed to test this plan as a behavioral experiment, she experienced new evidence that countered her 
prediction that she would be embarrassed, and instead experienced her teacher as helpful. New, positive 
experiences such as this one were able to build the child’s confidence in alternative responses to struggling 
with academics. 

Therapist Feedback in the Beck Initiative. 

Therapists participating in The Beck Initiative received feedback on their progress toward 
competency in cognitive therapy in two ways, each of which were intended to build new skills in an approach 
that echoes the CT approach with students. Small group meetings (5-7 therapists) were held weekly for two 
hours with audio review for at least 2 therapists per meeting. At the beginning of each meeting, an agenda was 
set collaboratively with the therapists. Prior to playing audio, therapists were asked to specify the feedback 
they sought for the cued audio. The therapist, his or her peers, and two Beck Initiative instructors then listened 
to segments of audio, provided verbal feedback, and suggested possible future directions. The instructors 
guided the structure of the verbal feedback to begin with reinforcement of skills demonstrated in the audio 
and then to the requested feedback. Before moving to the next agenda item for the meeting, the instructor 
confirmed that the therapist received the verbal feedback he or she was seeking and had a concrete plan of 
action for the next session. In many cases, therapists were given an opportunity to practice the indicated future 
suggestions by participating in role plays and receiving further relevant feedback from peers and instructors. 

In addition to verbal feedback in the weekly meeting, therapists received structured, 
individual written feedback at the midpoint (3-months post-workshop) and endpoint (6-months post 
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workshop) of the consultation group, using a gold-standard measure of competency. This written feedback 
was provided for each item of the competency measure (see below). 

Program Evaluation Measures 

Cognitive Therapy Competency. All 25 therapist trainees submitted audio recorded sessions with 
students that were rated using the Cognitive Therapy Rating Scale (CTRS; Young & Beck, 1980), an 11-item 
observer-rated measure designed to evaluate therapists’ overall and specific CT competencies, at three 
different time points: (1) prior to the workshop phase of training (baseline); (2) at the midpoint of the 6-month 
consultation phase; and (3) at the endpoint of the 6-month consultation phase. Each item is scored on a 7- 
point Likert scale, ranging from 0 (Poor) to 6 ( Excellent ); the 11 items are summed to yield a total CTRS 
score, ranging from 0 to 66. The CTRS contains two subscales: (1) general therapy skills (i.e., feedback, 
understanding, interpersonal effectiveness, and collaboration); and (2) CT-speeific skills (i.e., guided 
discover, focus on key cognitions and behavior, strategy for change, and application of CT techniques; V allis, 
Shaw, & Dobson, 1986). Traditionally, scores of 40 or above are considered to be indicative of competent 
delivery of CT (Shaw et al., 1999). In addition to the numerical ratings, raters provide trainees with narrative 
written feedback for each item of the CTRS, as well as general written feedback regarding the session as a 
whole. The written feedback on each item followed a general format of (1) describing what the therapist did 
well in the session, relevant to the item, and (2) suggesting specific changes the therapist could make or 
interventions the therapist could try to increase competency on the given skill. The CTRS has demonstrated 
adequate internal consistency and inter-rater reliability (Vallis et al., 1986), and strong inter-rater agreement 
for general competency (Williams, Moorey, & Cobb, 1991). 

CTRS ratings were conducted by the 2 instructors assigned to each agency. Prior to rating any audio, 
instructors were required to demonstrate that they were calibrated with gold-standard CTRS ratings. 
Instructors would listen to audio recorded sessions that had been previously rated by a group of calibrated 
raters. For an individual audio recording, the instructor “matched” the gold standard by (1) rating item scores 
within 1 point of the gold-standard rating for 9 out of 11 items, and (2) showing agreement with the gold 
standard about whether the total score indicated competency (>40) or non-competency (<40). For an 
instructor to be deemed calibrated, he or she must match on 3 out of 4 audio session rated. If the instructor did 
not match on 3 out of 4, a new set of 4 was completed as a new attempt. All raters in this study were 
calibrated prior to rating audio from trainees, and monthly calibration meetings were held to monitor and 
maintain reliability. 

Evaluation of Narrative Written Feedback on the CTRS. As mentioned previously, in addition to the 
11 quantitative items described above, the CTRS includes space for open-ended narrative feedback following 
each item. Coding of narrative feedback responses from the CTRS was conducted on the full sample (N = 25). 
A novel coding system was developed for the current study to describe the content of written feedback 
delivered to trainees via the CTRS. Narrative written feedback was organized into categories that were 
developed and clarified through several iterations. First, three postdoctoral fellows (1 instructor from the 
current study, 2 Beck Initiative instructors who did not participate in the school-based trainings) 
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independently read through the narrative feedback provided to five randomly selected participants and 
developed lists of all potential feedback categories. Via consensus, the categories were then synthesized to 
create a comprehensive and distinct list of 16 categories (see Table 1 for examples of each category): (la) 
positive observation, (lb) negative observation, (lc) neutral observation, (2) description of client behavior, 
emotions, or quotes, (3) positive statement followed by suggestion for improvement, (4a) suggestions to carry 
out, (4b) pitfalls to avoid, (5) normalization of therapist’s behavior, (6) pointers for raising a score, (7a) 
positive statement of growth, (7b) negative statement of growth, (8a) rationale for implementing a suggestion, 
(8b) rationale for refraining from a behavior or intervention, (9) implementation instructions, (10) positive 
exclamatory statement, and (11) miscellaneous. Next, the postdoctoral fellows operationally defined each 
category and trained two masters level research assistants to code each sentence of the narrative written 
feedback into one distinct category. During the coding process, the two raters flagged any sentences that did 
not fit into one precise category; based on consensus the fellows placed these sentences in one of the 
categories. A small number of responses were not fit to any category and were therefore placed in the 
miscellaneous category. 

In order to determine inter-rater reliability, both raters independently coded the narrative written 
feedback of three randomly selected participants. Reliability coefficients for each category, ranged from 
unacceptable (k = .16, Category 4b) to excellent (k = .90, Categories la and 3). For the subsequent analyses, 
two categories were combined (4a and 9, from here referred to as 4a\9) because the raters were unable to 
reliably distinguish them, indicating that they were describing the same construct. In addition, two reliability 
coefficients (Categories 7a and 7b) could not be calculated because the categories were coded with too low 
frequencies or not coded at all by at least one of the raters. Category 4b was excluded because the reliability 
was poor according to commonly accepted standards (Landis & Koch, 1977). All other categories (la, lb, lc, 
2, 3, 4a\9, 5, 6, 8a, 8b, 10, and 11) demonstrated at least moderate interrater reliability (k > 0.6) and were 
retained in subsequent analyses explained below and presented in Table 2. 

Knowledge of Cognitive Therapy. The Cognitive Therapy Knowledge Quiz is a 20-item multiple 
choice test designed to assess knowledge of CT principles and interventions (e.g., “According to the cognitive 
model, a person’s interpretations of life events lead to emotions and behavior”). The quiz was developed 
specifically for The Beck Initiative to test trainees’ knowledge of CT before (baseline) and after the 22-hour 
workshop (approximately 5 weeks post-baseline). Preliminary results of this measure have been presented 
previously, indicating that the Knowledge Quiz is sensitive to change in knowledge acquisition resulting from 
training (i.e., Corso, Cunningham, Sposato, & Buchhofer, 2010). A subset of the sample (n = 16) completed 
this measure. 

Program Feedback from Trainees. In an effort to evaluate trainees’ satisfaction with The Beck 
Initiative training program, participants were asked to complete an 11 -item anonymous program feedback 
form at post-workshop, mid-practicum (3 months post-workshop) and post-practicum (6 months post¬ 
workshop). The feedback forms, which were developed for this initiative, were administered by the 
community partners and were de-identified before being shared with the training instructors. Given that the 
present study is focused on the effectiveness of the entire training program, only data from the final written 
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Table I. Cognitive Therapy Rating Scale Feedback Categories and Examples 


Category 

Number 

Category Name 

Examples 

la 

Positive observation 

Nice job explaining the thought-feeling-behavior triangle! 

lb 

Negative observation 

Some parts of your agenda were a little vague. 

lc 

Neutral observation 

You also had him review the triangle when talking about his 
belief that things aren’t fair. 

2 

Description of client behavior, 
emotions, or quotes 

He is very concrete in his responses. 

3 

Positive statement followed by 
suggestion for improvement 

It’s great that you wanted her to see that people aren’t better 
than her, but sometimes you were trying to prove it to her, 
rather than have her uncover it, in her words. 

4a\9 

Suggestions to carry out 

Remember to put the homework check on the agenda in future 
sessions. 

4b 

Pitfalls to avoid 

One thing to watch for is trying to convince her or talk her into 
changing her thoughts. 

5 

Normalization of therapist’s behavior 

I could hear you working hard at making sure all of the CT 
pieces were in there, but what sometimes happens is that 
therapists focus so much on doing the CT components that they 
forget to do some of the empathy and warmth that comes so 
naturally. 

6 

Pointers for raising a score 

To improve this score more, you could have asked if he’s okay 
moving on to the next topic of discussion before transitioning. 

7a 

Positive statement of growth 

Fantastic job - you’re really making good progress. 

7b 

Negative statement of growth 

Guided discovery is a great area for growth for you to focus on. 

8a 

Rationale for implementing a 
suggestion 

Getting more specific agenda items will likely lead you to 
specific interventions that can give your client a sense that they 
are chipping away at their problems. 

8b 

Rationale for refraining from an 
intervention or behavior 

Often when the agenda items are vague or broad it can lead to a 
vague or diffuse therapy session without any intervention. 

10 

Positive Exclamatory statements 

Nice work! 

11 

Statements not accounted for by any 
other category 

Maybe we can have some practicum group discussions around 
how to modify questioning for very concrete youth. 


feedback form will be presented. A subset of the sample (n = 13) completed this form. The form consists of 
three items rated on 7-point Likert scales assessing the overall quality of the training and comfort in applying 
the skills trainees acquired throughout the training. Three forced-choice (i.e., yes/no/maybe) questions 
measure trainees’ beliefs about adequacy of time spent in group practicum, relevancy of topics to trainees’ 
clinical work, and whether practicum sessions were useful adjunctive training tools to the training workshop. 
An additional four items inquire about the structure of the practicum sessions, as well as the most and least 
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helpful components of the training. The final item provides trainees with the opportunity to share additional 
comments regarding their experience with the program. 


Table II. Cognitive Therapy Rating Scale Feedback Categories, Mean per Therapist and Frequencies 


Category 

Number 

Category Name 

Mean 

SD 

Frequency 

Percent 

la 

Positive observation 

30.28 

16.96 

757 

25.89 

lb 

Negative observation 

4.24 

5.16 

106 

3.62 

lc 

Neutral observation 

11.60 

21.50 

290 

9.92 

2 

Description of client 

20.32 

33.24 

508 

17.38 

3 

Positive statement followed by suggestion 
for improvement 

3.72 

2.73 

93 

3.18 

4a\9 

Suggestions to carry out 

31.6 

13.52 

790 

27.02 

5 

Normalization of therapist behavior 

0.20 

0.50 

5 

0.17 

6 

Pointers for raising a score 

2.84 

3.44 

71 

2.43 

8a 

Rationale for implementing a suggestion 

6.40 

4.75 

160 

5.50 

8b 

Rationale for avoiding a behavior or 
intervention 

0.68 

0.90 

17 

0.58 

10 

Positive exclamatory statements 

3.24 

2.92 

87 

2.77 

11 

Miscellaneous 

1.16 

1.43 

29 

0.01 


Note. Mean = mean times category occurred per therapist; SD = Standard Deviation; Frequency = total number of times 
each category was coded (out of 2929 total feedback sentences); Percent = percentage of each category when all rating 
scales are summed. 


Results 

Training Effectiveness 

Baseline scores on the CTRS ranged from 6 to 35 (M = 20.54, SD = 8.28), 3-month scores ranged 
from 21 to 50 (M = 33.04, SD = 7.31), and 6-month scores ranged from 19 to 54 (M = 41.12, SD = 9.43). As 
noted previously, a total score of 40 is widely accepted as the clinical benchmark of competency in CT. 
None of the participants reached a score of 40 or above at baseline, five participants achieved competency by 
the 3-month mark, and 18 participants demonstrated competency by the final assessment (72% of the 25 
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participants). In addition, participants are considered to be competent with regard to a specific skill if they 
receive a score of 4 or above on a given item. At baseline, the mean scores for individual items ranged from 
.63 (homework) to 3.13 (interpersonal effectiveness), suggesting that, on average, participants did not 
demonstrate competency on any of the skills assessed by the CTRS. In contrast, individual item means 
ranged from 3.36 (homework) to 4.2 (interpersonal effectiveness) at the 6-month point, indicating that, on 
average, participants approached competency on all items. 

Hierarchical linear modeling (HLM) growth curve analysis was used to assess changes in CTRS 
scores over time because it more accurately captures individual growth compared to traditional methods of 
repeated measures variance tests (Bryke & Raudenbush, 1992). The model predicting change over time 
within participants using CTRS total score as the outcome variable was significant ( Coefficient = 10.28, SE = 
1.10, t(24) = 9.39,p< 0.001). 

Knowledge of Cognitive Therapy 

A subset of participants (n = 16) completed the CT Knowledge Quiz at the beginning and end of 
training in CT. These participants demonstrated a statistically significant increase in their knowledge of CT 
principals and theory, f(15) = -2.758, p < .05. This change also represented a large effect (Cohen’s d = -0.69). 
The participants who completed this measure did not differ significantly in CT competency compared to 
those who did not complete this measure ('/( I) = .62, p = .43). 

Program Feedback from Trainees 

At the end of the practicum phase, 13 trainees (52%) completed a program feedback form developed 
specifically for this initiative. Therapists who completed the program feedback form did not significantly 
differ in CT competency from those who did not complete the feedback form (r/ 2 (2) = .65, p = .72). Given the 
small sample size, there was insufficient power to run statistical tests of differences between feedback 
completers and non-completers on all demographic measures. However, a non-significant tendency was 
identified for therapeutic orientation to predict feedback form completion (y 2 (4) = 8.98, p = .06). 

Overall, trainees’ feedback about the program was notably positive. The mean score for the overall 
quality of the practicum phase of training was 5.38 (SD = .77, range = 4-6) on a scale of 0 (Poor) to 6 
(Excellent). The mean ratings for the degree to which participants felt comfortable applying CT in general, 
and the extent to which participants felt comfortable applying the particular content discussed during the 
practicum phase were 4.54 (SD = .88, range = 3-6) and 4.92 (SD = .86, range = 4-6), respectively, on a scale 
of 0 (Not at all comfortable) to 6 (Very comfortable). Of the 13 trainees who provided feedback about the 
program, 98% (n = 12) stated that the amount of time spent in group practicum was adequate, and 85% (n = 
11) found the practicum topics to be relevant to their clinical work. All 13 trainees found the practicum 
sessions to be a useful, adjunctive training tool to the workshop. 

Participants were also asked to identify the most helpful component of the training. In response to 
this question, 77% (n = 10) of participants made reference to the feedback they received from their mentors. 
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For example, one trainee wrote: “The feedback and suggestions given each week were very helpful.” With 
regard to the least helpful components of the training, 38% (n = 5) of participants declined to comment. An 
additional 38% (n = 5) commented on the structure and logistics of the consultation sessions (e.g., “I believe 
the weekly sessions could have [been] a little longer”). Only one trainee commented on the quality of 
mentors’ presentation style (“sometimes it felt as though group discussions evolved into lectures...”). 

Evaluation of Narrative Feedback 

In order to gain a systematic understanding of the way in which mentors provide written feedback to 
trainees, we examined the frequency of the feedback categories described above that met our criteria for 
reliability. The most common type of feedback fell under the umbrella of suggestions to implement and step- 
by-step recommendations for how to do so (combined Categories 4a\9), which accounted for 27% of the total 
CTRS sentences. Positive observations (Category la, 26%) was the second most common category, followed 
by Category 2, which captured sentences describing students’ behaviors and feelings (17%), and neutral 
observations regarding the session (Category lc, 10%). Feedback providing a rationale for a specific 
suggestion to implement (Category 8a) accounted for 6% of CTRS sentences, whereas negative observations 
accounted for 4% of feedback sentences. The remaining categories each accounted for less than 4% of the 
total feedback (see Table 2 for the frequency of occurrences of each category). 

Discussion 

The program evaluation findings suggest that the Beck Initiative training model is a promising 
method for training school-based community mental health workers to deliver CT competently with students. 
Over the course of the training program, therapists were able to demonstrate the acquisition of cognitive 
therapy skills as evaluated by expert raters, and demonstrate those skills in the schools. Therapists 
demonstrated significant gains in skill by the mid-point of training, and they demonstrated further gains by 
the end of training, with almost three quarters of the trainees demonstrating competency at 6 months. 
Notably, no restrictions were placed on the treatment goals or presenting problems of students, which led to 
training cases representing the broad spectrum of students seen in school-based services. The majority of 
therapists were able to deliver CT competently with their students, even among the challenges inherent in 
delivering services in this high-demand, real-world setting. The success of this training suggests that training 
school-based community mental health service providers in CT or other evidence based practices may be one 
effective way of increasing access to care for youth who may be otherwise underserved. 

Consistent with this demonstration of skill, trainees also evidenced increases on the test of CT 
knowledge after completing the workshop. Prior to training, average knowledge test scores indicated that the 
baseline level of knowledge about CT was quite low. Scores increased after the workshop phase of the 
training, showing meaningful gains. In fact, the majority of the trainees scored a 90% or higher on the post¬ 
workshop test, indicating that they answered no more than 1 or 2 items incorrectly. This accumulated 
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knowledge may have set the stage well for the trainees to benefit from the 6 months of practiced application 
and feedback and suggests that this type of workshop may be an effective component in effective training. 

In addition to demonstrating increased skill and knowledge, trainees provided quite positive 
feedback about the training experience. Trainees particularly valued feedback pertaining to their emerging 
skills, underscoring the importance of the feedback component of the training program. Analysis of the 
content of written feedback at the midpoint and end of training revealed that specific suggestions and 
instructions were the most common form of written feedback provided, followed by the provision of positive 
feedback. Negative feedback occurred quite infrequently in the written comments. This focus on a strength- 
based approach to clinical training appears to have been effective, given that almost 75% of the trainees were 
able to demonstrate competency in CT after 6 months. These findings are a step toward quantifying 
successful and acceptable components in this training initiative, which in turn can guide further refinement 
of effective training programs. This unpacking of the components of the Beck Initiative may help guide 
future research to distill the most effective and acceptable approaches to train community therapists to 
competency, increasing overall community access to evidence based care. 

Although these findings suggest that the model of training employed in the Beck Initiative may be an 
acceptable and possibly efficacious approach to training school-based therapists in CT, the study has 
limitations that may impact the findings. The small sample size of therapists precluded the analysis of the 
relation between the content of the written feedback and trainees’ performance. The limited variability within 
the written feedback (largely instructional and positive) and training outcomes (largely successful in training 
therapists to competency) further limited the ability to analyze the relation between written feedback and 
training outcomes. The use of instructor ratings on the CTRS rather than blind raters may have resulted in a 
bias in competency scores or feedback, although this approach was necessary from a training perspective. 
Reliability of the raters was monitored and maintained throughout the study, which may have limited the 
impact of any bias. In addition, our findings are further limited by missing program evaluation data for a 
number of the therapists included in this sample, attributable to the voluntary nature of participation in the 
program evaluation. Without a control or comparison group, these data stand primarily as a preliminary 
indication that method of training was acceptable to therapists and an effective tool in teaching CT to school- 
based therapist. Only the written feedback was analyzed, and although it may serve as a proxy for the 
weekly verbal feedback, it cannot be assumed that the written feedback is fully representative of the briefer 
but more frequent verbal feedback. 

Given the rising call for empirically supported school-based interventions (Forman & Burke, 2008), 
the preliminary outcomes of the training model suggest that further examination of the efficacy and impact 
of this training may be merited. First, a more rigorous randomized study of the model would help to increase 
confidence in its effectiveness, including a larger sample size for better statistical power, a control group for 
comparison, measures of the impact of training on student functioning, and measures of the degree to which 
CT has been implemented in the school setting. After establishing the effectiveness of the training, a study 
of sustainability would provide important data about the degree to which investing in this CT training might 
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lead to long-term change in the services provided. A larger sample would also allow for examination of 
whether the content of feedback, whether written at the 3-month point or verbal in weekly groups, might 
predict competency by the end of training. Each of these further directions would facilitate a better 
understanding of factors that might enhance or inhibit the implementation of evidence based treatment in 
schools, ultimately leading to greater confidence in the services received by our students and better access to 
effective care. 

The Beck Initiative training in CT is one important step in answering the call for broad access to 
effective care. This training model was not limited by a manual or to a specific presenting problem, but 
instead used case conceptualization to meet the wide-ranging needs and strengths of students in their regular 
academic environments. Bringing evidence based practices to the schools where students spend their days 
may reduce the challenge of connecting children and adolescents with the services that can improve their 
lives, moving ever toward equal access to effective services for all. 
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